Surgical simulator user input device

ABSTRACT

A virtual surgery system or virtual testing system provides a simulation or test based on image data. A simulator combined with a real exam requires simulation tasks by a test taker. Additionally, a surgical procedure may be simulated using image data of a patient in devices simulating the physical instruments a surgeon uses in performing the actual procedure, for example. The user input device, such as a mouse, three dimensional mouse, joystick, seven dimensional joystick, full size simulator, etc., can be used in a virtual simulation to move through the image data while the user looks at the data and interaction of the input device with the image data on a display screen. Force feedback can be provided based on physical constraint models (of the anatomy, for example), or based on edge and collision detection between the virtual scope or virtual tool used by the operator and walls or edges of the image data in the image space. The virtual simulator may be used as a teaching, training, testing, demonstration, or remote telesurgery device, for example.

This is a division of U.S. patent application Ser. No. 08/412,805 filedon Mar. 29,1995 pending.

BACKGROUND OF THE INVENTION

The present invention relates to a virtual surgery system in which asurgical procedure may be simulated using image data of a patient anddevices simulating the physical instruments a surgeon would use inperforming the actual surgical procedure.

Various prior art training devices and devices used in surgicalapplications are described as follows.

U.S. Pat. No. 5,149,270 issued on Sep. 22, 1992 to McKeown discloses anapparatus for practicing surgical endoscopic procedures. The McKeownpatent relates to simulators that incorporate features to simulatevisual and manipulation surgical conditions for training surgeons insurgical procedures such as laparoscopy and hysteroscopy. The apparatushas a cavity in which an object simulating a human organ is mounted forperforming the practice procedure. The cavity is closeable to outsideview or access, thus forcing the individual to use and manipulate theinstruments under conditions that mimic real life operating anddiagnostic conditions. U.S. Pat. No. 5,149,270 is incorporated herein byreference in its entirety.

U.S. Pat. No. 4,907,973 issued on Mar. 13, 1990 to Hon discloses amedical investigative system in which a person interacts with the systemto interject information that is utilized by the system to establishnon-restricted environmental modeling of the realities of surrogateconditions to be encountered with invasive or semi-invasive procedures.This is accomplished by video display of simulated internal conditionsthat appear life-like, as well as by display of monitor data including,for example, blood pressure, respiration, heart beat rate and the like.U.S. Pat. No. 4,907,973 is incorporated herein by reference in itsentirety.

U.S. Pat. No. 5,273,038 issued on Dec. 28, 1993 to Beavin discloses acomputer system receiving two dimensional slice data of a heart or otherorgan to be simulated in three dimensions. This three dimensional imagedata and chemical composition data of the heart or other organ arestored in the computer memory. Then a Voxel View or three dimensionalvolume rendering program forms images of the organ to be studied.Diagnostic data obtained from a patient with electrical measurementsignals including an electrocardiogram, electro-myogram,electro-encephalogram, and other diagnostic measured electrical signalsobtained from a patient are fed into the system and are placed incomputer memory. Physiological data of the patient, including thestrength, weakness, and other parameters of the organ is also considereddiagnostic data and is supplied into the system This data may be fed inblack and white or in color to a device which shows the organ forvisualization, operation simulation, or training. U.S. Pat. No.5,273,038 is incorporated herein by reference in its entirety.

U.S. Pat. No. 5,222,499 issued on Jun. 29, 1993 to Allen et al.discloses a fiducial implant for the human body that is detectable by animaging system. The placement of the fiducial implant into a portion ofthe anatomy of the human body allows for the recreation of a particularimage slice of the portion of the anatomy taken by an imaging systemwith respect to a first time period, at subsequent imaging sessions andalso with different scan modalities. This provides a doctor with theability to accurately follow the progress of the portion of the anatomyof interest.

U.S. Pat. No. 5,385,474 issued on Jan. 31, 1995 to Brindle discloses amethod for simulating anesthesiology and operating room conditionsincluding the following six steps: displaying initial patient simulatedvital sign information from a memory to signify an initial patientcondition; randomly modifying the displayed patient vital signinformation according to a script matrix in a manner analogous to thatin which a patient's vital signs would be effected in the operating roomby drugs or other external effects, thereby indicating a deterioratingcondition; displaying user options; evaluating the timeliness andappropriateness of user input selections from the options in response tothe changes in patient vital sign information to improve its initialstate or deteriorate to a critical state in accordance with thesuccessive script blocks in the script matrix depending upon the user'sresponse and timeliness.

U.S. Pat. No. 5,261,404 issued on Nov. 16, 1993 to Mick et al. disclosesa three-dimensional mammal anatomy imaging system and method whichprovide images of the internal anatomy of a mammal. U.S. Pat. No.4,261,404 is incorporated herein by reference in its entirety.

U.S. Pat. No. 4,331,422 issued on May 10, 1994 to Loftin et al.discloses a training system for use in a wide variety of training tasksand environments. Artificial intelligence is used to providecomputer-aided training.

U.S. Pat. No. 5,343,871 issued on Sep. 6, 1994 to Bittman et al.discloses a method and apparatus for mediating a biofeedback sessionwith a human subject in which measurements of electrophysiologicalquantities are used to control a presentation to the subject of a seriesof prestored audio-visual sequences designed to induce a desiredpsychological state when viewed.

U.S. Pat. No. 5,130,794 issued on Jul. 14, 1992 to Ritchey discloses apanoramic image based virtual reality display system.

SUMMARY OF THE INVENTION

The present invention relates to a virtual surgery system simulator orvirtual testing system simulator providing a simulation or test based onimage data. The test simulator can include test authoring, test taking,assessment, training, demonstration, etc. The present invention alsorelates to a simulator combined with a real exam which can requiresimulation tasks by a test taker. In a virtual surgery embodiment of thepresent invention, a surgical procedure may be simulated using imagedata of a patient and devices simulating the physical instruments asurgeon would use in performing the actual procedure, for example. Imagedata is stored in the memory of the computer which corresponds to aportion of an anatomy in a three dimensional data set. A user inputdevice such as a mouse, joy stick full size simulator mock-up, etc. isused to move through the image data while the user looks at the imagedata on a display screen. A virtual surgery or other virtualimplementation can be simulated based on the image data and the movementof the input device.

In embodiments of the present invention, force feedback may be providedbased on a collision detection between the virtual scope or virtual toolused by the operator and walls or edges of the image data in the imagespace. Physical constraint models may be used for the force feedback.Alternatively, edge detection and collision software may be used todetect a collision between the virtual scope or tool and the walls oredges of the image data. Force feedback is then performed in response tothe edge and collision detection using mechanical, electromechanical,pneumatic, hydraulic, etc. type devices. The present invention may alsobe used as a teaching, training, testing, demonstration, etc. device.Demonstrations may be provided to a user of the simulator before, duringor after the simulation. The testing mode provides questions before,during and after the testing simulation and requests the user to performspecific tasks as well as using text questions. The task questionsrequire the user to go through the virtual image data in a particularmanner using the scope or other virtual tool device and can determinewhether or not the test taker is correctly performing the operation. Inthe tutorial or teaching mode the present invention provides feedback tothe user such as the quality of performance of the user or givinghelpful hints to the user to perform the required tasks.

Virtual surgical operations can be performed according to an embodimentof the present invention using recorded expert simulations or real-timetutorials via a video or other data feed line. Telesurgery can beperformed using a simulator according to the present invention in whicha surgeon performs functions using a virtual mock-up of surgicalinstruments while a robot in a remote location performs the actualsurgery based on the surgeon's movements relating to the virtualsurgical devices. This implementation of telesurgery could also be usedin other applications in which a simulator is implemented to performspecific tasks or actual remote tasks are performed using a simulatorwhile a robotic device performs the actual task based on the simulatedmovements.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features and advantages of the present invention will becomeapparent in view of the following description taken in conjunction withthe attached drawings.

FIG. 1 illustrates a virtual surgery system according to an embodimentof the present invention.

FIG. 2 illustrates a virtual mouse and virtual orifice and boxconnection which may be used in an embodiment of the present invention.

FIG. 3A illustrates a top view of a virtual mouse according to anembodiment of the present invention.

FIG. 3B illustrates a left side view of a virtual mouse according to anembodiment of the present invention.

FIG. 3C illustrates a right side view of a virtual mouse according to anembodiment of the present invention.

FIG. 4 illustrates a portion of a box device near a virtual orifice usedto accommodate various inputs such as a hose used for virtual surgery.

FIG. 5 illustrates an embodiment of the present invention in which aloop of flexible hose is accommodated within a box for instructionalpurposes.

FIG. 6 illustrates a virtual reality surgical simulator which may beused in a kiosk form and which includes physical constraint models.

FIG. 7 illustrates a joystick having seven degrees of freedom for aninexpensive implementation of surgical (or other) simulation.

FIGS. 8A and 8B illustrate a further embodiment of a joystick accordingto the embodiment of FIG. 7.

FIG. 9A and FIG. 9B illustrate a deflection of a universal ball jointportion of the bottom of the joystick shaft of FIGS. 7 and 8A.

FIG. 10 illustrates a box structure in which a position of a mouseinside the box structure is determined based on laser or ultrasounddeflection.

FIG. 11 illustrates a pin input/output device which can input or outputdigital image information corresponding to output/input physicaltextures pushed against the pins.

FIG. 12 illustrates a pneumatic or hydraulic feedback device accordingto an embodiment of the present invention.

FIG. 13 illustrates a flow chart corresponding to a program implementedin the processor or computer according to an embodiment of the presentinvention.

FIG. 14 illustrates an embodiment of the present invention which relatesto a "mouse-in-a-mouse", or a"virtual-instrument-in-a-virtual-instrument" or "multiple virtualorifice" implementation of the present invention.

FIG. 15 illustrates a real surgery implementation of the presentinvention at a mouth orifice of a patient.

DETAILED DESCRIPTION

FIG. 1 illustrates a virtual surgery system according to an embodimentof the present invention. The virtual surgery system includes a computer100 including, for example, a processor 102 and a memory 104. Thevirtual surgery system additionally includes a virtual scope handlemouse device 106, a tube or endoscopic shaft 108 and a box device 110.The mouse device 106 corresponds to a real surgical instrument with agrabber, knife, laser, catheter, etc. on the working channel of thescope (or mouse). The tube 108 may be, for example, a tube, shaft,column, solid or hollow, flexible or stiff, with end deflection or noend deflection, and is inserted in a virtual orifice 112 of the boxdevice 110. The virtual orifice may correspond to a mouth, nose, anus,or other orifice of the anatomy, or an incision in the anatomy. Feedbackis provided from one or more of the mouse 106, the tube 108 and thevirtual orifice 112 via lines 114, 116 and 118, respectively.Additionally, other feedback may be provided from the inside of box 110to computer 100 via line 120. The computer can also control thesevarious elements of the virtual surgery system using lines 114, 116, 118and 120 (e.g., to implement force feedback, which will be described infurther detail below). Further, the computer 100 is not necessarilydedicated only to the virtual surgery or other simulation. Computer 100and/or processor 102 can also perform other functions or procedures inaddition to the simulation described herein. That is, computer 100 canbe a general purpose non-dedicated computer performing and/or able toperform a wide variety of tasks in addition to those corresponding tothe virtual surgery simulation.

Image data is stored in memory 104 of the computer 100. This image datamay be data corresponding to a portion of a human anatomy or an animalanatomy, or an entire human or animal anatomy. The image data stored inmemory 104 may be a three dimensional data set obtained, for example,using image data using ultrasound techniques, or from a CT scan, a PETscan, an MRI scan, etc. Alternatively, the image data stored in memory104 may be obtained using other various human anatomy image data (orother image data). For example, the National Library of Medicineprovides image data of the entire human anatomy of a frozen cadaverwhich has been sliced, imaged and digitized. This visible human projectimage data is available through the National Library of Medicine, andmay be obtained, for example, on the Internet at: ACKERMAN @ NLM.Gov.

Another example of a method of obtaining three-dimensional anatomy imagedata is provided in U.S. Pat. No. 5,261,404, which has been incorporatedherein by reference.

Memory 104 may also store image data unrelated to or having no bearingon a human or animal anatomy. This image data may relate to particularvolumes, shapes, sizes, textures, etc. This unrelated image data may beused in a virtual simulation where a user is learning how to use thescope prior to performing any virtual surgery simulations as describedherein. This simulation is similar to a virtual surgery simulation, butoperates on the unrelated image data.

In implementing a virtual surgical procedure, a mouse device 106corresponding to the particular surgical procedure to be implementedwith a virtual surgery system is attached to the tube 108 at point 107.A variety of mouse devices 106 may be attached at point 107 to allow avariety of surgical procedures using the virtual surgery system. Imagedata of, for example, the generic human anatomy or data corresponding toa particular patient are then retrieved from memory 104 and displayed ondisplay device 130. Memory 104 stores data including, for example, adata set library of abnormalities, infections, etc. The data set libraryof abnormalities, infections, etc. may be related to a particularpatient or medical information in general, such as tumors, infections,scans, Xrays, etc. For example, the data in memory 104 could includeimage data corresponding to a patient having bronchitis, a variety oftumors, or other conditions, foreign objects, etc. (either normal orabnormal).

The particular patient image data can be chosen by the surgeon prior tothe virtual surgery operation by an input device connected to computer100. For example, a keyboard device (not illustrated in FIG. 1) can beattached to computer 100 to allow a user to choose a particularoperation to be performed or particular image data to be used in thevirtual surgery procedure. Alternatively, the user can make the choiceusing other devices such as a touch screen on the display 130 or using amouse device to choose a particular operation or image data currentlydisplayed on the display 130. The surgeon is then able to do a "practicerun" on a particular patient using the virtual surgery system prior tothe actual surgery of the patient, for example. The virtual scopeincluding the mouse device 106 and the endoscopic hose 108 can be usedto navigate through a three dimensional image model. Further details andembodiments of the system illustrated in FIG. 1 and other virtualsurgery system embodiments will be discussed below in reference to otherdetailed drawings of embodiments of the present invention. The virtualsurgery system according to the present invention device can access animage data set of the anatomy to traverse, surgery to perform, etc. andcan interpret the input device as that from a variety of differentsurgical instruments.

A summary of a patient's case, including history, physical exam, lab andXray, vital signs and any other clinical data, for example, may also bestored in memory 104, displayed on display 130 simultaneously with asimulation being displayed on display 130, either before, during orafter the surgery simulation. This data may be displayed on a splitportion of the display, or in a small window portion of the display,etc. If the simulator is used in conjunction with a real surgery, thatdata can also be stored and/or displayed in addition to the simulation,before, during, or after the actual procedure or simulation. The imagedata and other clinical data corresponding to a patient can also bestored on a computer disk, for example, and placed in a patient'smedical record for later use or review, if necessary.

FIG. 2 illustrates a virtual mouse and virtual orifice and boxconnection which may be used according to an embodiment of the presentinvention. For example, the mouse, box and virtual orifice connectionillustrated in FIG. 2 may be used in the overall virtual surgery systemof FIG. 1. The same reference numerals as those of FIG. 1 are used inFIG. 2 to illustrate the same features of the present invention.

The virtual mouse 106 of FIG. 2 includes, for example, switches (orbuttons, handles, etc.) 130, 132 and 134. These switches can performvarious functions based on the endoscopic mouse or virtual scope beingused to implement a particular surgery. For example, switch 130 may beused as an anterior-posterior end deflector which actually deflects theend of the tube 108 in a direction 140, 142, 144 according to theposition of the switch 130 on the virtual mouse device or giveinformation to the computer simulating such movement in a real surgicalinstrument either hand-held or robotic in a telesurgical operation.Switches 132 and 134 can be used, for example, for suction or airinsufflation. The tube 108 may be a generic tube which can be rigid orflexible. Arrows 150 and 152 illustrate directions in which the virtualmouse scope 106 and hose 108 may be moved by a surgeon in performingvirtual surgery. Arrow 150 illustrates that the mouse 106 and hose 108may be moved in an up and down direction, while arrow 152 illustratesthat the mouse and hose may be moved in a twisting or rotating motion.The mouse 106 and hose 108 could also be moved in x, y and z directionsas well as pitch, roll, yaw and the rotating (or twisting) direction.Although the bottom of hose 108 is illustrated as being bent or bendable(or deflectable) in the embodiment of FIG. 2, other embodiments of thepresent invention can be implemented in which the bottom of hose 108 isnot bent bondable deflected or deflectable. For example, an embodimentof the present invention may be used in which the hose is a straighttube used as an optical endoscope with no bending or deflection of theend of the tube.

A mouse device 106 which may be used in implementing the presentinvention and attached to a virtual surgery system similar to the mouse106 in FIG. 1 and FIG. 2 is illustrated in FIG. 3A, FIG. 3B and FIG. 3C.FIG. 3A illustrates a top view of the endoscopic mouse 106, FIG. 3Billustrates a left side view thereof, and FIG. 3C illustrates a rightside view. The mouse 106 includes switches 202, 203, 204, 206, 208 and210. The mouse 106 illustrated in FIGS. 3A, 3B and 3C is an endoscopicmouse/virtual scope to be used in a bronchoscopy (using, for example,switch 204) or gastroscopy (using switches 206 and/or 208) type ofoperation. Switch 202 provides suction, switch 204 provides airinsufflation, switch 206 is an anterior-posterior deflector, switch 208is a lateral deflector and switch 210 is an anterior-posterior enddeflector. Flexible shaft 212 of the mouse 106 is the flexible shaftportion which connects to the hose 108 at point 107. A variety ofdevices 106 could be placed at the end 107 of the hose 108 to model allsorts of surgical instruments or even tools used in other trades. Thatis, the present invention need not be limited to virtual surgeryapplications. For example, the virtual system of the present inventionmay be implemented in an industrial tool environment such as using adrill press, or in landscape architecture, building architecture,manufacturing, plumbing applications, sewage clearing, disasters innuclear plants, jet or automobile engines, outer space applications,robotic devices, etc. This is advantageous in a virtual device in whicha particular tool was attached to the hose 108 to simulate a particularprocedure in which an instrument is attached to a tube-like hose such ashose 108 which is inserted into some other structure to perform aparticular procedure.

FIG. 4 illustrates a portion of the box device 110 near the virtualorifice 112 which is used to accommodate various inputs such as the hose108 used for virtual surgery. The virtual orifice 112 can accommodatevarious sizes of inputs and various inputs such as a rigid scope hose, aflexible scope hose or other tube-shaped devices. The tube 108accommodated by the box 110 can correspond to various actual medicaldevices such as any endoscope, endotracheal tube, nasogastic tube, IV's,catheters, etc. The tube 108 is accommodated within the box 110 viarollers 230, 232, 234 and 236, arms 238 and 240, and springs 242 and244. The arrangement of rollers 230, 232, 234 and 236, arms 238 and 240and springs 242 and 246 moves in the direction of arrows 246 and 248based on the size of the tube 108 being accommodated within the virtualorifice 112 and box 110. Additionally, the arrangement can be moved inthis direction to provide force feedback to the hose. Spring tension canbe increased by way of a solenoid pinching pressure on the rollers. Oneor more of the rollers 230, 232, 234 and 236 can also be fitted withstepper motors or friction devices to allow application of force. Thecomputer (not shown in FIG. 4) can also be programmed to provide forceas determined by the parameter being simulated, and a computer detectionof resistance to movement or an encountering of a collision between, forexample, the hose (or virtual scope device) with a virtual structurewhich is firm or has elastic properties.

FIG. 5 illustrates an embodiment of the present invention which uses aloop of flexible hose 108 which is accommodated within the box 110. Theuser of the virtual system can feed the hose 108 through the box 110 inan up and down or rotating, etc. manner similar to that described above.The hose may be pushed into or pulled out of the box 110, twisted,rotated, etc. This embodiment may be used as a training device forfeeding a surgical hose through a virtual orifice 112. The virtualorifice may be adjusted (e.g., by computer programming or physicaladjustment) to be any orifice of the body (e.g., mouth, nose, anus,etc.) or a puncture through the skin into a blood vessel. The additionalcorresponding anatomy would then follow. For example, when dedicated asthe mouth, the anatomy encountered would be the adjacent structures suchas the pharynx, larynx, esophagus, then stomach, etc.

Tactile feedback may be provided to a mouse device such as an endoscopicsimulator by attaching physical constraining models in which an inputdevice is manipulated. As an example, the physical constraining modelmay be a portion of the bronchial anatomy, the gastrointestinal anatomy,urologic, or other portions of the anatomy such as walls of a bloodvessel. The physical model constraints are constructed to beapproximately the same size as the virtual computer model correspondingto image data stored in the memory of the computer. The physicalconstraining models may be used to provide tactile feedback as the hose(or endoscopic device) hits the physical walls of the physical model andthe image walls on the computer image. This is an inexpensive way ofproviding tactile feedback without acquiring edge detection or collisiondetection software programs to determine when the mouse device or hosemeets or collides with an edge or wall in the image data. Additionally,this method provides force feedback without any expensive steppermotors, magnets or other constraining devices required in the mousedevice itself. FIG. 6 illustrates such a typical model constraint devicein which virtual orifices are used through which a scope is traversed ina virtual surgery simulation. The scope is positioned over anappropriate physical constraint model for purposes of simulation so thatthe simulator can perform a variety of different types of surgicalsimulations.

FIG. 6 illustrates a virtual reality surgical simulator which may beprovided in a kiosk form. The kiosk 300 includes a housing 302 whichcontains the computer 100 (not illustrated in FIG. 6) and the display130. The kiosk 300 includes a side door 304 which can open as shown byarrows 306. Alternatively, door 304 could be a sliding door which slidesstraight out from the side of the housing 302. The door 304 is a largebox-like structure which includes physical constraint models 310, 312and 314. The physical constraint models used in this embodiment can eachbe a set of connecting tubular structure each representing a differentportion of the internal anatomy. These physical constraint models cancorrespond to, for example, the lungs, the stomach, or the urologicportions of the anatomy, etc. Virtual orifices 316, 318, 320 and 322 areprovided at the top portion of the door 304 of the kiosk 300.Alternatively, the virtual orifices could slide over the appropriateconstraining model and snap into place with a physical latch and/orelectrical connection with the computer to identify which portion of theanatomy is to be simulated. The virtual mouse 106 and hose 108 may beinserted through one of the virtual orifices 316, 318, 320 and 322 sothat the endoscopic hose 108 is accommodated within the physicalconstraint models 310, 312, 314, etc. The physical constraint models caninclude any types of physical constraints corresponding to a portion ofthe anatomy which is to be surgically probed.

The physical constraint models 310, 312, 314 can be built as a physicalmodel corresponding to different portions of the anatomy. The physicalconstraint models may be constructed using stereo lithography by takinga three-dimensional data model and shining lasers to heat a plasticplasma surface and provide a complex cast of the particular portion ofthe anatomy based on a physical computer model constructed fromthree-dimensional image data, as well as by more conventional means ofmodelling. When the surgical instrument (e.g., the hose 108) hits a wallof the physical constraint model 310, 312, 314, a device such as amagnet, actuator, solenoid, spring, etc. is used by the feedback systemof the computer to provide force feedback. The force feedback device isconnected via lines 114, 116, 118 and 120 of FIG. 1, for example,between the physical constraint model and the computer 100.

The kiosk 300 can also include a credit/access card swipe portion 330which may be used to access the device using a magnetic card or a creditcard to charge a user for use of the kiosk or virtual surgery system.

Other embodiments of the kiosk 300 of FIG. 6 could be implementedaccording to the present invention. For example, an embodiment couldinclude a scope 106 separate from the simulator kiosk 300 . In such anembodiment, a user walks up to the kiosk and plugs in his/her personalscope or scopes, for example. In another embodiment, a communicationlink to the simulator such as a telephone or satellite link may be usedin combination with a virtual scope, joystick, etc. in a remotelocation. A demonstration (or simulation) can thus be performed on thedisplay of the kiosk from a remote location. Alternatively, a remotehome computer can be used at which the user moves the virtual scope orjoystick, etc., and uses the processing power of a powerful computersuch as an IBM mainframe computer, for example.

FIG. 6 illustrates a kiosk including a door device having a plurality ofvirtual orifices at the top. Alternative embodiments of the presentinvention include a box device permanently, temporarily, removably,exchangeably or detachably attached to a floor, wall, or table andhaving one or a plurality of virtual orifices therein. A portable boxwhich is removably attached to a table, wall, or floor, or not attachedat all, may also be used in implementing embodiments of the presentinvention.

In an embodiment of the present invention in which a virtual simulationis used in a testing environment, a signature pad or other device foridentifying a test taker can be used to identify the test taker.

While FIG. 6 illustrates an embodiment of the present invention in whichphysical constraint models are used to detect a physical actuation of adevice against an edge of the physical constraint model, other methodsmay be used to detect the edge of the virtual scope hitting an edge of avirtual anatomy. For example, if physical constraint models are not usedwithin the box 110, virtual models may be used by implementing knownedge collision and detection software, such as the Telios programavailable from High Techsplantations of Bethesda, Md. or EngineeringAnimation, Inc. (EAI) of Ames, Iowa. Force feedback may then be providedfrom the edge collision and detection software running in the computerbased on the image data to a force feedback device (e.g., magnet,solenoid, actuator, spring, etc.).

FIG. 7, FIGS. 8A and 8B, FIG. 9A and FIG. 9B illustrate a "joystick" orother desktop input device for a computer for a virtual realitysimulation device. Alternatively, the "joystick" can be used in playing,for example, realistic three-dimensional video or computer games.

In the joystick illustrated in FIG. 7 and FIGS. 8A and 8B, a shaft on aplunger is mounted inside an axle-like device which can be rotated andwhich is mounted in the frame in a pivotable manner. The joystick allowsleft and right, up and down, in and out, or x, y, z orientation, andallows inputs in a positive and negative direction for each variable.The handle of the joystick device may include switches or devices wherethe thumb and fingers grip the device for additional input from theuser. A harness which holds the axle holding the piston can be mountedon a rotation device so that the rotation of the entire structure can beadded as another coordinate of input movement (i.e., the rotationaldirection). The "joystick" or universal input device can be used as athree-dimensional mouse or to simulate a virtual surgery operation suchas surgical an operation requiring endoscopic instrument positions.

FIG. 7 illustrates a joystick having seven degrees of freedom for aninexpensive implementation of surgical (or other) simulation. In usingthe joystick of FIG. 7, a surgical simulator may be provided which doesnot require the large kiosk illustrated in FIG. 6 and which can beimplemented in, for example, a computer or video game implementation.Even if the joystick of FIG. 7 is not used in a computer gameimplementation, it may be used for other virtual surgeryimplementations. The joystick may also be used for other simulationimplementations or general computer games which include seven degrees offreedom as illustrated in FIG. 7. Specifically, most joysticks havepreviously allowed movement in two directions (i.e., the x andy-directions). The joystick illustrated in FIG. 7 allows one to use ajoystick having a better virtual movement in three dimensions.Specifically, the joystick of FIG. 7 allows movement in the x, y, z,pitch, roll, yaw and rotation directions.

Joystick 400 of FIG. 7 includes a handle having attached switches ordials 404. The shaft 406 allows movement in the plus/minus Z directionsas illustrated by the arrow at the top of FIG. 7. Disks 408 and 410,which are attached to the top and bottom of shaft 406, respectively,provide a stabilization when the handle 402 is moved in a rotating orcircular direction. Walls 400 of the joystick 400 include protrusions414 which act in cooperation with disks 408 and 410 to maintain thejoystick in a stable position upon the rotation thereof. Additionally,walls 412 and protrusions 414 may be used to measure the rotation ofdisks 408 and 410 to determine the amount of rotation in the joystickand provide electrical signals to the computer 100 relating to therotation. The universal ball joint 416 allows a pivoting motion in theholder 418 to allow for movement in the x and y directions. Spring 420allows up and down movement in the Z direction by allowing the verticalmovement of the handle. A spherical base 422 allows pitch, roll and yawof the joystick device.

A further embodiment of the joystick device according to the presentinvention is illustrated in FIGS. 8A and 8B. Electrical signals are fedback from different portions of the joystick such as the mouse handle402, the disks 408 and 410, the spring 420, etc. Strain gages, springdeflectors and other devices used to measure the quantities and thesignals are sent to the computer using electric wires or infrared radiofrequency or other technology. The joystick is mechanically returned toa neutral position after being moved by a user when force is no longerprovided to the handle by the user. Handle 402 can be detachablyreplaced by other handles such as scalpel 430 as shown in FIG. 8B.

Further features of the joystick devices of FIGS. 7 and 8A and 8B areillustrated in FIGS. 9A and 9B. FIGS. 9A and 9B illustrate thedeflection of the rounded portion 416 of the bottom of the joystick andthe spring 440 or other device (such as a cam, leaf spring, weightedbottom, magnet, etc.) which is used to return the joystick shaft to theupright neutral position. As illustrated in FIG. 9B, when the shaft 406is tilted in a certain direction, the spring 440 actually returns theshaft to a neutral position, unless a user is continually applying aforce to the handle 402 of the joystick device. Upon deflection of theshaft 406, a device such as gauge 450 may be used to determine thedeflection thereof. Alternatively, electrical signal 452 may be providedfrom the spring to the computer 110 to determine the deflection.Additionally, in order to provide force feedback, a magnet, motor,solenoid, pinchers, constrictors, stepper motors, electromagnetics, conesqueezer, physical constriction along an entire length of the shafts,etc. may be used. The force feedback is provided when the joystick shaft406 is moved to a point where a collision occurs with the image datastored in the memory of the computer. This is determined, for example,using collision and detection software.

In an alternative embodiment of the present invention illustrated inFIG. 10, a mouse 502 uses lasers or ultrasound emanating from a portion504 of mouse 502 to detect a position of the mouse based on a deflectionof the laser or ultrasound off the walls of a room or box 510 in whichthe mouse is accommodated. A grid may be provided on the inside walls ofbox 510 to determine the relative position of the mouse 502 therein. Inthis manner, three dimensional virtual surgery or other virtual realitymay be implemented using a mouse which is freely moveable within thespace of the box 510.

In the embodiment of FIG. 10, an array of electric eyes shining at eachother, for example, in the left to right and up to down directionswithin the box or room 510 may be used which shine light at each otherto determine a position of the mouse 502 in the box or room 510. Laserscan be used in this embodiment to shine from one side of the box to aphotosensor, for example, on the other side of the box in order to scanthe space within the box 510 and thereby interpret the reflection of themouse 502 or transmission to the other side of the box to determine theposition of the mouse. Alternatively, ultrasound technology may be usedin which proximity sensors are placed on portions of the walls of thebox or at opposite sides of the room or outside the box or room todetermine a position of the mouse device. Standard mathematicalalgorithms may be used to determine a position in physical space of themouse in the room or box in the emobiment of the present inventionillustrated in FIG. 10.

FIG. 11 illustrates a simulated surface input/output device as describedbelow. A plane 602 having a plurality of pins 604 thereon is used forphysical modeling and digitization of image data. The pins 604 are ofcolumnar shape with sensors and actuators or solenoids attached to thepins. FIG. 11 shows alternative embodiments in which a solenoid 606, astepper motor 608, an ultrasound device 610 or a pressure gauge 612 areconnected to each of the pins. Any of these devices provide input to acomputer 614 via either infrared or connected lines as illustrated inFIG. 11. The pins of FIG. 11 and corresponding solenoid 606, steppermotor 608, ultrasound 610 and pressure gauge 612 operate in the manneras described below.

An input/output device may be used according to an embodiment of thepresent invention for physical modeling and digitization of image datasuch as in the embodiment of FIG. 11. The input/output device includes aseries of pins 604 of columnar shape with sensors and actuators orsolenoids attached to the pins. The pins 604 are arranged in a tightarray, either in a single pin depth and a long line or grid withmultiple rows in each direction. The pins 604 may be used as an outputdevice by receiving image data from the computer and moving the pins ina position corresponding to the image data values. The pins 604 may beused as an input device by pushing a physical product such as, forexample, an artificial hip prosthesis into the pins and recording in thememory of the computer as image data the positioning of the pins basedon the physical product being input. This would measure one surface at atime. Each individual pin 604 represents an x and y location in acomputerized grid. The z dimension in the grid would be represented bythe position of the pin 604 after the device came into contact with theproduct being recorded. This results in a crude digitization of a simpleobject or a complex object. No actuating device is required as part ofthe input device. The device could also detect a position of the pin 604and move the pin so that each pin 604 is extended out of the surface toa predetermined level based on the digitized model input to thecomputer. In order to make the texture of the tops of the pins smooth, asurface material can be laid over the pins.

The input/output device of FIG. 11 could be used as an input in asurgical virtual reality system according to an embodiment of thepresent invention. Additionally, the pin input/output device could beused to input simple or complex shapes for rapid digitizing, includingoutputting and illustration in Braille text, reproducing maps for theblind, architectural modeling, a medical/physical exam teaching devicesuch as teaching self-examinations to define tumors and superficial bodyorgans (for example, breast or testicle), or modeling of physicalappearance for sculpture of forensic medicine.

In a preferred embodiment, the pins 604 of FIG. 11 could be fiber opticin nature. Such an arrangement could facilitate the study of light onthree-dimensional objects to allow computer calculation of issues suchas the solar gain on a new building or how well an audience might see aperformance based on the construction of a stadium.

Additionally, the fiber optic pins would allow the pin array to be verytight. If each of the pins is flexible and enough room was allowed forthe advancing and retracking of the pins, the surface of the pins couldhave an image projected thereon through the fiber optics or by aprojector yielding a three-dimensional visual image for athree-dimensional T.V. or a realistic colorized model product. If thematerial covering the end of the pins is a diffuser of light, the lightand image of the overall input physical device could be merged into oneso that a clear image can be seen rather than an image including manysmall points of light.

FIG. 12 illustrates a pneumatic or hydraulic feedback device forimplementing force feedback according to an embodiment of the presentinvention. The scope handle or mouse device 702 is attached to a shaft704. A plunger 706 is attached to the bottom of the shaft 704. The shaft704 protrudes through a virtual orifice 708 and into a housing (orcylinder) 710. A valve wheel with stepper motor control is identified byreference numeral 712. A pressure valve 714 provides a pneumaticfeedback signal to computer 718 when an object hit occurs (e.g., whenthe virtual scope runs into a wall or other portion of the image data).The computer determines an object hit or collision using collisiondetection software, for example. Upon an object hit, the pressure valve714 closes and provides the pneumatic feedback signal 716 to computer718. Computer 718 provides a signal 720 to the valve wheel with steppermotor control 712 to provide the equivalent of a bumpy or rough textureupon turning of the valve wheel 712. This provides force feedback to theplunger 706, shaft 704 and scope handle 702. Infrared signals 732 and734 are provided to the computer 718 from the scope handle 702 and thevirtual orifice 708, respectively.

Force feedback in the embodiment of FIG. 12 is accomplished using astepper motor with a screw type pneumatic cylinder. The valve wheel canbe adjusted based on hydraulics or pneumatics, etc. Forward motion by auser on the scope handle moves the piston or shaft 704 freely. Whenforce is applied, the pressure valve 714 closes on the exit cylinder,thus causing pressure to build against the piston. When a certain forceis reached, the piston stops. If the user continues to push on the scopehandle, a threshold is reached and "perforation" occurs and pops off avalve release on the pressure valve (not illustrated in FIG. 12). Thedepth of the plunger (or shaft) into the cylinder 710 corresponds to asimilar depth in an organ of the body (e.g., the colon). The forceapplied by the force feedback provides a similar force as that requiredfor a medical object to advance further into the organ.

Similar resistance to removal can then be accomplished measuringnegative pressure. For example, in removing a portion of the anatomywith the scope biopsy instrument, resistance will be met until thetissue pulls away from its initial location.

FIG. 13 illustrates a flow chart corresponding to a program implementedin a processor or a computer according to an embodiment of the presentinvention. In particular, a computer program corresponding to the flowchart of FIG. 13 may be implemented by computer 100, processor 102 orcomputer 718, for example. Further, certain steps of the flow chart ofFIG. 13 are optional in a program implemented according to an embodimentof the present invention. Further, additional embodiments of the presentinvention may be implemented which are not particularly illustrated inthe computer flow chart of FIG. 13 as described elsewhere herein.

The programs of the flow chart of FIG. 13 starts at step 802. In step804, a model of an anatomical space is received by the computer orprocessor. Textures are applied to visible surfaces of the anatomicalspace model in step 806. In step 808, physical characteristics areapplied to surfaces (e.g., distensibility, elasticity, tensile strength,etc.). As an example, an abnormality may be added to the image data(e.g., image data representing a tumor). A shape, size, proportion, etc.relative to the space in which it is placed may be chosen. Optional step809 inputs a test writing module. The test writing module allows textquestions and task questions to be asked of the test taker. The testtaking embodiment is described elsewhere herein, but relates to activityof moving to a location in an anatomy, for example, or performing arequired procedure, etc. A Medical Examiner Board, for example, candetermine questions to be used based on previous responses or actions bytest takers, etc.

Data compression is performed in step 810. This data compression couldcorrespond to averaging, video sequencing or other techniques used toreduce the data load. Alternatively, data compression may not benecessary when medical illustration data or other artist rendition ofanatomical image data or other image data are input in the program. Adata set of the three dimensional image data space is obtained by theperformance of the steps mentioned above. Modifications may be made tothe three dimensional image space. Additionally, reference portions maybe added to the image space using a training module, a case designmodule, data corresponding to a test being performed by the virtualsurgery system including text or tasks to be performed by the user, orreference to literature references such as medical or other references,on-line medical data, etc. While steps 804-812 are not necessarilyperformed each time a new simulation is started, these steps may beperformed at any time to enter new image data into the computer or newtest information or questions.

In step 814, a particular procedure to be used by the user of thevirtual surgery system is chosen by a user or a certain test isimplemented by the program. A user may pick an anatomical set from thememory, an instrument or instruments to be used by the virtual scope,and other data pertinent to a desired procedure to be implemented. Uponbeginning a new procedure in 814, the program of FIG. 13 performsinformation such as recording information in a personal filecorresponding to a particular user, writing multiple text, playback orprint to video tape reports corresponding to the virtual surgery to beperformed, etc. Step 816 tracks the location of an instrument orinstruments being used in the virtual surgery procedure. This trackingof step 816 is performed in response to electronic feedback informationfrom the mouse device, virtual orifice, box, or hose from the virtualsurgery system. Step 816, in tracking the location of the instrument orinstruments, can determine what the optics at the end of the hose islooking at in the image data, record all movements and locations in theimage data, etc. Step 818 performs an optional interaction withreference materials such as, for example, medical texts, files,electronic simulations, other texts, text books, journals, literalsearches of journals, still photographs or other digital or non-digitalimages, motion images, cartoons, other simulations, video, audio (suchas a voice announcing location or consultation, either recorded orreal-time), references, tests, images, demos, real-time consultation,recorded information, etc. A hypertext linked to references may be madein relation to the location, other text, an abnormality, or a mistake inuse of the instrument or judgment of the user. Step 820 calculates theinteraction of the virtual instrument with the image data. For example,in a virtual surgery system including more than one virtual scope device(e.g., where a second input device such as biopsy forceps are placedinto the scope), image manipulation of the data set may be made throughthe scope using one or more of the various instruments used in thevirtual surgery. The calculating step 820 additionally can monitor aninput device such as a keyboard, mouse device, scope device, orifice,joystick, full size instrument, second, third or other pluralinstruments, a button, switches, a handle, etc.

Step 822 performs edge and collision detection according to standardedge and collision detection software. This edge and collision detectionstep 822 calculates interaction with surfaces and provides signalscorresponding to feedback information such as force feedback. Step 824performs force feedback to a user in response to the collision detectionstep 822 using electromechanical, magnetic, etc. devices. Steps 822 and824 are optional and are not necessary, for example, if a device such asa physical constraint model is used in implementing the presentinvention.

Step 826 displays the image data, location of the instrument, and anyinteraction of the instrument with the image data in an area near thecurrent position of the image data. Once a specific location of theanatomy is reached which has been input by the user or required to befound by the user by a test device according to the present invention,text, an image or a request to perform a particular task may bedisplayed on the display device. This is particularly helpful in theoptional examination mode or optional "tour guide" mode, with text,video or image data displayed. Another optional tutor may be used withmultimedia demos, for example, from a CDROM or data stored in a memoryto tutor a user on how to perform a particular operation eitherrequested by that user or requested by the virtual surgery system.

Steps 828 determines if the simulation has ended. If it has not ended,program flow returns to step 816. If it has ended, step 830 thendetermines if a new simulation or test procedure is to be performed(either in response to a user input or a requirement of acomputer-controlled test, demonstration, etc.). If a new simulation isto be performed, flow returns to step 814. If not, a report is producedin step 831 and step 832 then ends the simulation. The report may relateto a particular experience by the user, a comparison with correctresponses, how the simulation may be changed or customized based onoverall results, etc.

FIG. 14 illustrates an embodiment of the present invention which relatesto a "mouse-in-a-mouse", or a"virtual-instrument-in-a-virtual-instrument" or "multiple virtualorifice" implementation of the present invention. A first virtual scopeor mouse device 902 is attached to a hose 904 at an end portion 906thereof. The hose 904 extends through a first virtual orifice 908 and abox device 910. The hose 904 extends into the box device 910 similarlyto the hose 108 extending into the box 110 in the embodiment of FIG. 1.A first virtual orifice 908 is attached at a top portion 912 of the boxdevice 910. In the embodiment of FIG. 14, a second instrument 914 suchas biopsy forceps is attached to a shaft 916. The shaft 916 extendsthrough a second virtual orifice 918 which is included in the firstinstrument (or mouse device) 902. Signals from the mouse device 902, thebiopsy forceps 914 and/or the first and second virtual orifice 908 and918, among other locations, is provided to a computer such as computer100 illustrated in FIG. 1. The shaft 916 extends through the virtualorifice 918 and into the mouse device 902. As illustrated in FIG. 14,the shaft device 916 extends into the hose 904 past the end portion 906of the mouse device 902. In an alternative embodiment, the shaft 916could extend into the mouse device 902 through the second virtualorifice 918 without extending into the hose 904, as illustrated by thedotted line shaft 916A.

The biopsy forceps 914 (or other second instrument) included in anembodiment of the present invention as illustrated in FIG. 14 areinserted through a handle of the first virtual scope 902 through thesimulated second orifice 918 of the scope 902. In a preferredembodiment, the virtual shaft 916 is extended through the virtual scope902. The device could have a handle on either mouse device 902 or biopsyforceps 914, for example, to simulate cutting, biopsying, laser work, orother activity. As discussed above, the shaft 916 can exit through thehandle or a hollow channel in the shaft of the scope, or into the hose904. This allows a simulation of a multiple instrument surgicalprocedure used by an operator through the simulated scope and first andsecond orifices. Force feedback may also be performed according to theabove discussion.

Biopsy forceps 914 could alternatively be replaced by a syringe handle,scissors, or other instrument appropriate for a particular operation. Asmentioned above, the embodiment of FIG. 14 provides a scope within ascope or mouse within a mouse implementation of virtual surgeryaccording to an embodiment of the present invention. Shaft 916 operatesas the working handle and provides operational switching electronicallyhooked to the computer (not illustrated in FIG. 14).

In implementing the multiple virtual scope device illustrated in FIG.14, a user can manipulate the first virtual scope 902 through the firstvirtual orifice 908 to get to a particular location. The second virtualscope (or biopsy forceps) 914 is then added via shaft 916 for cutting,biopsy, laser, or other work to be performed in a virtual operation.Manipulation of the biopsy forceps 914 moves both of the virtualorifices illustrated in FIG. 14. In this manner, the multiple mouse orvirtual scope embodiment simulates a standard endoscopic biopsytechnique or other use of multiple tools in a channel of a scope toprovide virtual scope within a scope or mouse within a mouse simulation.

FIG. 15 illustrates a real surgery implementation of the presentinvention at a mouth orifice of a patient. FIG. 15 illustrates anembodiment of the present invention in which actual surgery data may beobtained by a computer 1002. A virtual orifice 1004 is attached over areal orifice of a patient 1006 (in the embodiment of FIG. 15, the realorifice is the patient's mouth). The virtual orifice 1004 is attached tothe patient 1006 using a device to hold the virtual orifice 1004 inplace, such as cloth ties 1008. A real endoscope 1010 is then used by asurgeon to perform real surgery on the patient 1006. The endoscope isplaced through the virtual orifice 1004 and into the patient's mouth toperform the real surgery. The motion of the endoscope 1010 within thepatient's mouth is tracked by the computer 1002, which receives signalsfrom the virtual orifice 1004, the end 1012 of the real endoscope 1010,the handle 1014 of the real endoscope 1010, etc. For example, thesequence and time in each region of the anatomy may be recorded and/orthe anatomy may be re-created. The motion during the real surgerytracked using this embodiment may be merged with CT scan data or otherthree dimensional data, merged with images from the scope, or mergedwith any other image data. Alternatively, a simulation may be mergedwith real video, pictures or image data obtained during the realsurgical procedure (e.g., from the real endoscope or virtual orifice).This creates a marked improvement in documenting an actual procedure fora patient's medical record. This data may be stored in a memory of thecomputer 1002, or in any other recording device such as a CDROM,magnetic tape, computer disc, etc. The saved data may be used tore-create an actual surgery (e.g., in forensic medicine, in legalcourtroom for medical malpractice purposes, for teaching purposes, orfor review of a surgeon's work by the surgeon's superiors, the surgeon'speers and/or a Medical Review Board or Medical Examination Board, etc.).Further, the computer may take the stored data tracing the motion of thesurgery, either alone or with the merged ultrasound scan, CT scan, MRIscan, PET scan, etc. and provide the information to a recording device(video tape, CDROM, etc.). The recording device may then be forwarded toa virtual surgery system such as in the embodiment of FIG. 1 (e.g., viacourier or U.S. Postal Service) to be used as an instructional device,for example. The image data may also be provided electronically viaInternet, modem, etc. to a virtual surgery system according to anembodiment of the present invention.

In a virtual surgery system according to the present invention asdescribed above, as a user moves a virtual scope to traverse through animage space and perform virtual surgery in that image space, thecomputer or processor tracks where the endoscopic camera is, and whatthat camera is looking at. In this manner, the position of the virtualscope arrangement and objects caught by the camera image in the imagespace are determined by the computer. The virtual surgery systemaccording to the present invention may be used as a simulator includingtesting, test altering, test taking, assessment, training, etc.purposes. A user could go to the virtual surgery system and chose aparticular virtual surgery to be performed, or alternatively could entera demonstration mode or testing, etc. mode. In a mode in which thecomputer acts as a tutor or "tour guide", the user may be asked certainquestions by the virtual surgery system, either by a question displayedon the display device or even a question posed to the user through, forexample, a speech synthesis device. The virtual surgery system can tellthe pupil or test taker where the virtual scope or input device islocated at in a portion of the anatomy or can ask the test takerquestions or require the test taker to traverse through the image spaceto a particular portion of the anatomy or perform a particular surgicaloperation on a portion of the anatomy. The tutor device can also requirethe user to perform a particular surgery or other procedure to a certainportion of the anatomy and can give feedback relating to the user'sperformance. For example, the virtual surgery system could tell the userthat the device is in a particular portion of the anatomy or give theuser instructions on how to get to a particular portion of the anatomy.Feedback result data may be provided to a pupil, user, or test taker viaaudio, video or text on the display screen. A text box, audio file orvideo file may be used in which a demonstration is performed by thesimulator showing the student how to perform the particular requiredoperation. The simulator can show a user how to perform the operationcorrectly or a picture of an image of the portion of the anatomy towhich the user is attempting to locate using the simulator.

An inexperienced doctor or student could use the tour guide beforeperforming a new procedure, then practice the procedure using thesimulator, and eventually perform the real procedure (e.g., a realsurgery on a patient or other real procedure). Additionally, thedoctor's or student's simulation could be stored in the computer forlater review by the doctor's or student's superiors and/or teachers,etc.

The virtual surgery system of the present invention may also be used inrelation to teleradiology, telesurgery, telemedicine, telemonitoring,etc. Teleradiology digitizes and provides image data for a particularpatient. This digitized and imaged data can be used as an input to thevirtual surgery system and stored in the memory of the computer, forexample. The medical information or image data can also be provided tothe virtual surgery device via a modem, telephone, satellite, etc.connection providing analog or digital data to be stored in the virtualsurgery system for later use. Additionally, patient monitoringinformation can be stored in the virtual surgery system for later use invirtual surgery.

Telesurgery may be used according to an embodiment of the presentinvention in order for a surgeon to perform surgery from a distance, orto provide consultation to another surgeon performing a real operationby an expert surgeon watching the real operation and instructing thedoctor using a simulation of a similar operation, etc. In an applicationin which a surgeon performs surgery at a remote location, a robot can beused to simulate hand movements of the surgeon at the remote locationvia a tele-robotic unit. The robot can move the real endoscope or othersurgical device according to the movements of the surgeon performedusing a virtual scope. In another embodiment of the present invention, asurgical procedure can be simulated by an expert surgeon, for example,in a library tutorial provided on a video tape, CDROM, electronic devicesuch as a modem, etc. Alternatively, the simulated procedure can beprovided by one surgeon to another surgeon at a remote location inreal-time using a video data feed. For example, a surgeon using a realendoscope looking at a real patient and moving the endoscope inside theorifices of a real patient, can receive video corresponding to datatransmitted electronically to a remote point (e.g., from the Mayo Clinicor via the Internet), and an expert watching the operation in real-timecan show the actual doctor performing the real surgery a simulation ofthe operation or provide particular guidance to the other surgeonperforming the real operation. This guidance can be provided on adisplay screen in the actual operating room while the surgeon isoperating on the actual patient.

A storage library can be implemented according to an embodiment of thepresent invention in which a library of image data simulations, problemsencountered, etc. are stored for later retrieval by a student orsurgeon. For example, an expert surgeon holding a simulator can simulatea biopsy or how to use a laser or particular surgical device on apatient with a particular abnormality or operation to be performed. Thesimulation library can also include particular image data correspondingto a specific patient. Such image data corresponding to a patient can bestored as a part of that patient's medical record, for example.

The present invention can also be used in a tele-robotics applicationfor other implementations other than virtual surgery. For example, avirtual industrial application or other virtual procedure can beimplemented according to an embodiment of the present invention.

Force feedback may be performed in implementing the present inventiondescribed herein using a spinning stepper motor, pressure, pinchers,constrictors, electromagnetics (e.g., when the scope is coated with amagnetic material and a magnet is used to create friction or resistanceto the scope), or variable tension using electromagnetics, pneumatics,hydraulics, etc., or other electronics such as solenoids, etc. Aconstriction device which may be used according to an embodiment of thepresent invention for force feedback which uses cone-shaped squeezers tosimulate constriction of the scope. Alternatively physical constrictioncan be provided along an entire length of the simulator.

A virtual surgery system according to an embodiment of the presentinvention can be used in which an input device is used by a user toperform virtual surgery or virtual medical testing or virtual testing ofother industrial applications as described above. The input device usedin implementing the present invention has been described herein as beinga mouse device, a seven dimensional joystick device, a full sizesimulator, etc. The input device used according to embodiments of thepresent invention can include a keyboard, a standard mouse, a threedimensional mouse, a standard joystick, a seven dimensional joystick, ora full size simulator with a full size mock-up of a medical or otherindustrial type instrument. Additionally, any of these input devices canbe used in the present invention with force feedback being performed.Physical force feedback can be implemented using physical constraintmodels or edge and collision detection software techniques.

Other embodiments of the present invention can be used in which, forexample, a patient has image data scanned into the system, and during asimulation or a real surgery operation, a portion of the display screenshows a pre-recorded expert simulation via video tape, CDROM, etc., or areal-time tutorial by another doctor. Telesurgery applications may alsobe used as described above, in which a surgeon moves an input device(e.g., a full-size virtual scope or instrument) of a simulator while arobot actually performs a real operation based on the simulated motionsof a surgeon at a remote location. The present invention may be used ina testing embodiment in which the virtual surgery device or othertesting device questions via text and specific task questions. Forexample, in a medical embodiment, the virtual device might ask a testtaker to go to a particular location in the anatomy and then perform abiopsy. Questions may be inserted in the test before, during or after aparticular operation (such as a bronchoscopy). A multitude of tasks maybe required of a student during the test procedure. The test taker maychose between different modes, such as an illustration, practice or exammode. As a result of students tests, reports may be issued relating tothe experience a particular student had during the test, how well theydid, in comparison to the correct procedures with the individualsperformance, and an indication of the performance of all individualstaking these tests for a particular question. In this manner, an examcan be determined and customized for a particular company, for example.In another embodiment, the Medical Examination Board can identifydifferent test questions by case, one time individual performance,cumulative performance by an individual, etc., and can provide differentlevels of difficulty. The virtual surgery system of the presentinvention or other test taking device not related to surgery or medicalapplications can include training, test taking and records archivingabilities (for example, in a medical context this archiving can relateto a patient's medical records).

In a medicine simulation embodiment of the present invention, volumes,surfaces of volumes, various characteristics and images of surfaces maybe merged with real images or illustrated images using mathematicalalgorithms. Algorithms may also be used to determine the intersection ofsurfaces in the image set with the particular test taking device.Simulation through the image data may be made while savingdocumentation, questions and other information relating to themanipulation of the image volumes. In another embodiment of the presentinvention, multimedia demonstrations may obtained from the manufacturerof a particular instrument or a manufacturer of an instrument differentfrom the instrument being used. If a surgeon or other user is using anew instrument, a demonstration may be made at the time of use (forexample, in an operating room), without even using a simulating device.The surgeon or other user could then use the new real instrument inresponse to touching the touch screen on the simulator device, forexample. In this manner a reference file is accessible at a point ofuse. In another embodiment, the touch screen of the simulator can beused to get on-line help from a manufacturer, medical director, etc., oran order for such help or a particular product may be made immediatelyusing the touch screen. References, experts, catalogs, products,demonstrations on how to use the product, etc. could be referred toeither in real-time, using prerecorded messages or ordering commands viathe simulating device or other device such as a touch screen.

What is claimed is:
 1. A surgical simulator user input devicecomprising:a virtual medical instrument corresponding to a real surgicalinstrument used in a medical operation, said virtual medical instrumentconnected to a computer for simulating a surgical procedure andincluding a virtual mouse attached to a tube; wherein said virtualmedical instrument is capable of simulating multiple real surgicalinstruments by interchangeably utilizing a different virtual mouseattached to said tube wherein each of said different virtual mouseperforms functions associated with a particular real surgical instrumentto be simulated.
 2. The surgical simulator user input device accordingto claim 1 further comprising a housing representative of a virtualpatient wherein said virtual medical instrument extends inside saidhousing through a virtual orifice and wherein said virtual orifice isconnected to a computer for providing data to the computer forsimulating a surgical procedure.
 3. The surgical simulator user inputdevice according to claim 2 wherein the computer provides data to saidvirtual orifice for simulating a surgical procedure.
 4. The surgicalsimulator user input device according to claim 3 wherein said dataprovided to said virtual orifice by the computer adjusts said virtualorifice to represent a particular orifice of a human body.
 5. Thesurgical simulator user input device according to claim 2 furthercomprising means for providing force feedback to said virtual medicalinstrument.
 6. The surgical simulator user input device according toclaim 5 wherein said means for providing force feedback on said virtualmedical instrument is edge collision and detection software resident inthe computer used for conducting a virtual surgery procedure and a forcefeedback device in said housing.
 7. The surgical simulator user inputdevice according to claim 5 wherein said means for providing forcefeedback is rollers, arms and springs.
 8. The surgical simulator userinput device according to claim 5 wherein said means for providing forcefeedback is a mechanical device.
 9. The surgical simulator user inputdevice according to claim 5 wherein said means for providing forcefeedback is an electro-mechanical device.
 10. The surgical simulatoruser input device according to claim 5 wherein said means for providingforce feedback is a pneumatic device.
 11. The surgical simulator userinput device according to claim 5 wherein said means for providing forcefeedback is a hydraulic device.
 12. The surgical simulator user inputdevice according to claim 2 wherein said housing includes rollers, armsand springs and wherein said virtual medical instrument is accommodatedwithin said housing by positioning said virtual medical instrument bysaid rollers, arms and springs.
 13. The surgical simulator user inputdevice according to claim 1 wherein said virtual medical instrumentincludes a hand-operable activation device for simulating a particularprocedure of a simulated surgery.
 14. The surgical simulator user inputdevice according to claim 13 wherein said hand-operable activationdevice is an electrical switch.
 15. The surgical simulator input deviceaccording to claim 13 wherein said hand-operable activation device is amechanical actuator.
 16. A surgical simulator user input devicecomprising:a first surgical instrument simulator module for simulatingprocedures performed in a surgical procedure; and a second surgicalinstrument simulator module for simulating procedures performed in asurgical procedure; wherein said first surgical instrument simulatormodule and said second surgical instrument simulator module areinterconnected and wherein said surgical simulator user input device iscapable of simulating multiple real surgical instruments byinterchangeably interconnecting different first and second surgicalinstrument simulator modules wherein each of said surgical instrumentsimulator modules perform functions associated with a particular realsurgical instrument to be simulated.
 17. A surgical simulator user inputdevice according to claim 16 wherein said first surgical instrumentsimulator module is a virtual mouse and said second surgical instrumentsimulator module is a tube.
 18. A surgical simulator user input deviceaccording to claim 16 further comprising a housing representative of avirtual patient wherein said second surgical instrument simulator moduleextends inside said housing through a virtual orifice and wherein saidvirtual orifice is connected to a computer for providing data to thecomputer for simulating a surgical procedure.
 19. The surgical simulatoruser input device according to claim 18 wherein the computer providesdata to said virtual orifice for simulating a surgical procedure. 20.The surgical simulator user input device according to claim 19 whereinsaid data provided to said virtual orifice by the computer adjusts saidvirtual orifice to represent a particular orifice of a human body. 21.The surgical simulator user input device according to claim 18 furthercomprising means for providing force feedback to said second surgicalinstrument simulator module.